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ABSTRACT 

This manual, developed in an effort to take the 
mysticism out of program evaluation, discusses six phases of the 
program evaluation process. The introduction discusses reasons for 
evaluation, process and outcome evaluation, the purpose of the 
handbook, the evaluation process, and the Sequoia United School 
District Dropout Prevention Program. Phase One provides a checklist 
for identifying the elements to be evaluated. An example is provided 
which illustrates the use of the checklist. Phase Two summarizes the 
steps used in producing evaluation questions for a program. A 
checklist is included which can be used as a guide. Phase Three 
provides a description of evaluation designs and data analysis 
procedures which could be used with these designs. Phase Four 
includes three categories to illustrate the instruments used for data 
collection. The steps to follow in selecting or constructing data 
collection instruments are summarized in a checklist. Phase Five 
discusses efficient collection, analysis, and interpretation of 
evaluation data. A checklist is included. Phase Six discusses the 
distribution, publication, and presentation of the evaluation report. 
Sections that should be included in an evaluation report are 
described, and a checklist is included to use as a guide in preparing 
evaluation reports. Illustrative examples and sample surveys are 
included. Throughout the manual , the Sequoia United School District 
Dropout Prevention Program is used as an example, and evaluation 
plans for interactive counseling services are included. (LLL) 
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FOREWORD 



Educators today are focusing on restructuring their school systems and educa- 
tional programs to respond to the current and changing needs of students. One 
component of the restructuring process that is generally agreed upon is the need for 
greater accountability and credibility. As educators, we have been quick to adopt new 
programs, materials, and delivery systems without evaluating the impact of these 
changes upon the student population for whom they were designed. Such lack of 
accountability has led to a gap in the credibility of our educational systems across the 
nation. 

It is time for educators who are in leadership and decision-making positions to 
welcome the opportunity to evaluate their programs ratherthan fearthe threat of failure. 
An evaluation designed to measure the goals and objectives of a particular program 
offers us opportunities to fine tune our programs and increase the chances of student 
success. 

The evaluation process should help program staffs more effectively design, 
develop, implement, and improve their programs. There a r e numerous reasons for 
evaluating our dropout prevention programs including: 

• to measure the strengths and weaknesses of the program and 
implement the appropriate changes; 

• to measure student outcomes; 

• to establish credibility for the program; 

• to provide a rationale for continuation of funding or system support; and 

• to document the process of program implementation for replication. 

The National Dropout Prevention Center has developed this manual in an effort 
to take the mysticism out of program evaluation. As educators interested in our youth 
who are at risk of dropping out of our schools, we need to design programs that meet 
their needs. Evaluating those programs is the only way we can know if we indeed have 
met that challenge. 



Dr. Nancy L. Peck, Associate Director 
Southeastern Desegregation Assistance Center 
Southern Education Foundation 
Miami, Florida 



Editor's typfe: 
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Dr. Pock is one of Iho founders of The National Dropout Prevention Network and served as the first chair of the Network's 
Executive Board. She also has been instrumental in the development and direction of the Center. 

I 



7 



The Evaluation Handbook 



ABOUT THE AUTHORS 



Dr. Peg L. Stank 

Evaluation Design and Staff Training Consultant 



Dr. Stank was the Division Chief for Research and Evaluation of the Pennsylvania Department of 
Education. She was responsible forthe design and implementation of evaluation procedures for statewide 
programs. She also served as the project evaluator for the Pennsylvania School Improvement Program 
(PSIP) funded by the National Institute of Education. PSIPwas designed to demonstrate and evaluate how 
research information was utilized in local schools. Dr. Stank now serves as Director of Research for the 
West Shore Institute in Harrisburg, Pennsylvania and as a certified trainer for Education Associates, Inc. 
in Frankfort, Kentucky. 

Her areas of expertise include program evaluation design and implementation which encompasses 
instrumentation, data collection, data analyses and reporting. Also, Dr. Stank has developed and 
conducted a wide variety of staff training programs, staff evaluation procedures and program monitoring. 



Dr. Jay Smink 

Executive Director of the National Dropout Prevention Center 



In 1988, Dr. Smink joined the National Dropout Prevention Center at Clemson University from the 
National Center for Research in Vocational Education at The Ohio State University where he was the 
former Director of Marketing and Dissemination. He earned his doctorate in Educational Administration 
from The Pennsylvania State University. His experience in research and program evaluation began as 
Director of the Research Coordinating Unit for Vocational Education in the Pennsylvania Department 
of Education. He was also the director of the Bureau of Research and Evaluation and at this time 
directed a national school improvement project featuring the utilization of new research products in local 
schools. He is also a member of a number of professional organizations which include: Phi Delta 
Kappa, the National Education Association, and the American Vocational Education Research 
Association (in which he has held office). 

Dr. Smink's 33-year career includes a strong background in public education programs at state and 
local levels. He has authored many articles for journals and newsletters as well as professional 
monographs and papers regarding research into and improvement of basic skills and vocational 
education programs He is a national leader and authority on dropout prevention, vocational education 
research and development, marketing, project management and evaluation. 



ERJ.C 



The Evaluation Handbook 



EVALUATION: AN INTRODUCTION 



WHY EVALUATE? 



Evaluation is a required, integral part of every successful dropout prevention program. A well- 
planned evaluation provides information that shows whether or not a program is operating as it was 
designed to operate; how well a program is functioning; and the impact of a program on the students, 
teachers, or other participants. 

The groups that require evaluation information about a program are the program decisionmakers, the 
funding agencies, the sponsoring institution's administrators, advisory groups, community agencies, social 
service agencies participating in the program operation, parents and the general public, as well as future 
clients and participants. These groups expect the program to be successful and want evidence to support 
their belief in the value of the program. 

The evaluation process will provide information that can be used to: 

• modify or adapt program operations; 

• justify program continuation and expansion; 

• support continuation of funding; 

• generate reports that may be used for public relation purposes; and 

• justify program termination. 
The evaluation process focuses on: 

• the stated goals and objectives of the program; 

• the critical elements and activities of the program as they are described in the proposal 
or program materials; and 

• additional areas requested by funding agencies, administrators, community agencies, 
social service agencies, businesses, and other groups participating in the program. 

The responsibility for evaluation rests with the program director. It is up to the director to plan and 
implement the evaluation procedures so that the decisionmakers will have evaluation information and 
reports for use in planning the future operation of the program. This does not mean that a program director 
will do the evaluation, but it does mean that the director will locate persons with the expertise to design and 
implement an appropriate evaluation procedure forthe program. Sometimes internal staff persons will be 
assigned responsibilities for evaluation procedures, and in othercases it may be necessary to hire external 
evaluation a factors. 
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EVALUATION PHASES 



There are specific phases in the evaluation process. First, the program elements or activities to be 
evaluated must be identified and evaluation questions generated for each of them. For each evaluation 
question, an evaluation design must be selected, data collection and analysis processes identified, and the 
dates for evaluation reports set. The data must be collected, analyzed and summarized, and the evaluation 
reports must be written. Within each evaluation phase there are many subactivities to consider when 
establishing the timeline for the evaluation process. 

Evaluation designs, data collection, and data analysis procedures o?n range from complex and 
sophisticated to very simple. Some designs require control or comparison groups with random sampling 
and data analyses such as Analysis of Variance Multiple Linear Regression, and Analysis of Covariance. 
This handbook will not deal with this level of sophistication— it will provide a straightforward description of 
evaluation using simple data analyses to describe the effects of a program on participants and the efficiency 
of program operation. 



PROCESS AND OUTCOME EVALUATION 

There are two types of evaluation neeaed to document the successes and weaknesses of dropout 
prevention programs: 

1 . Formative or process evaluation which addresses two major questions: 

• Are we doing what we said we were going to do in the proposal or program description? 

• Is the program operating efficiently and in a timely manner? 

2. Summatlve or outcome evaluation which addresses two major questions: 

• How well are the program's goals and objectives being met? 

• What is the effect of the program on participants? 

Process and outcome evaluation activities usually overlap, and data are often collected simultaneously. 
Process evaluation data summaries and reports are prepared during the program operation and may be 
used to adjust its operation for effectiveness and efficiency. The outcome evaluation data are not collected 
until the end of each cycle of proguim operation. 



PURPOSE OF THE HANDBOOK 

This handbook will provide a nontechnical description of each phase of evaluation and will provide 
sample forms and examples for adaptation and use by program directors and evaluation coordinators. For 
descriptions of more sophisticated approaches to evaluation, the coordinator should consult publications 
such as Kosecoff and Finks Evaluation Basic published by Sage, and other similar resource books. 
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The evaluation plan for a dropout prevention program should apply evaluation processes to (1 ) the 
school climate, (2) the community environment in which the program operates, (3) the major educational 
components of the program, and (4) the interaction among the school climate and the community 
environmental factors and the educational components, School climate refers to the overall conditions 
present in the school and community that influence the school program in positive and negative ways. For 
example, the relationship between the district administrative staff and school instructional staff may be very 
strained as a result of a recentteacher strike over a salary contract dispute This relationship would not prove 
to be beneficial for the initiation of a new project. Community environmental factors refer to the conditions 
that exist in the community related to school issues such as an upcoming school bond issue or a proposed 
new busing plan. Evaluation efforts shouid take these factors into account. 

The illustrative examples of the evaluation process given in this handbook will focus on one educational 
component of a fictitious dropout prevention program. However, in the real world the evaluation process 
would be applied to both the educational program components as well as the school climate and community 
environment in which the program operates. Exhibit l illustrates how a dropout prevention program must 
function within the constraints of many different school climates and community environmental factors. 



! EXHIBIT ONE: SCHOOL CLIMATE AND COMMUNITY ENVIRONMENTAL FACTORS 



SCHOOL CLIMATE FACTORS 



SCHOOL ADMINISTRATORS & STAFF RELATIONSHIPS 



DISTRICT MISSION & 
PHILOSOPHY 




PROGRAM GOALS 
& OBJECTIVES 



SCHOOL 
POLICIES 



CURRICULUM & 
INSTRUCTIONAL 
STRATEGIES 



COMMUNITY 
VALUES & 
PERCEPTIONS 



PUBLIC 



RELATIONS & 

COMMUNITY 

INVOLVEMENT 



FINANCIAL SUPPORT & 
BOARD STRUCTURES 



PARTNERSHIP 
PROGRAM 



SOCIAL SERVICE AGENCIES INVOLVEMENT 



COMMUNITY ENVIRONMENTAL FACTORS 
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A dropout prevention program has many other major components and specific elements that all need 
to be reviewed and evaluated. Examples of broad-based program components include: 

• staff selection and development, 

• public relations, 

• advisory committees, 

• parental involvement, 

• coordination of partnerships, 

• curriculum selection or development, 

• assessments and testing, 

• reporting accomplishments and failures, and 

• project management and scheduling. 

THE EVALUATION PROCESS 

There are six phases of the evaluation process presented in this handbook. 

PHASE I IDENTIFICATION OF THE PROGRAM COMPONENTS, 

ELEMENTS, PROCEDURES AND ACTIVITIES TO BE EVALUATED 

PHASE i! GENERATION OF EVALUATION QUESTIONS 

PHASE III SELECTION OF EVALUATION DESIGN AND PROCEDURES FOR 
DATA COLLECTION AND ANALYSES 

PHASE IV INSTRUMENTATION 

PHASE V DATA COLLECTION, ANALYSIS, SUMMARIZATION AND 
INTERPRETATION 

PHASE VI PREPARATION OF THE EVALUATION REPORTS 

Each phase of evaluation will be described and then a checklist of the steps within the phase will 
be displayed. The checklists will be followed by an example that illustrates the application of that 
phase of evaluation. All of the examples are based upon the fictitious dropout prevention program 
wnich follows and is described throughout the text of this handbook. 

1 ° 
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Sequoia United School District personnel initiated a dropout prevention program in 
September, 1981. The target at-risk population was ninth grade students identified by teachers 
and counselors from school academic, attendance, and disciplinary records. From 1975 to 1981 
the district had a dropout rate of 25 percent It was expected that a substantial decrease in the 
dropout rate would occur in the 1985 class because these students wot Id have b een served by the 
new program during all four years of high school 

The 1981-85 Dropout Prevention Program included a work-study partnership program with 
participating private sector employers, academic support services, and special inservice for staff 
and administrators. The district central office had actively pursued cooperative participation of 
community and social service agencies in the dropout prevention effort The evaluation 
information collected from 1981-1985 showed an increased community and business participa- 
tion in the program, successful work-study student experiences and an improvement in at-risk 
students' academic grades. However, the impact of the program on the graduation rate of the at- 
risk student was much less than expected. The program director met with school administrators 
and staff, program advisory groups, businesses participating in the work-study partnerships, 
participating community agencies, and participating social service agencies to discuss ways to 
improve the effectiveness of the Dropout Prevention Program. 

The evaluation information shared with these groups showed that over the four years of the 
program's operation the community groups and social service agencies had moved from an 
awareness of the dropout problem to active participation in the program activities. The academic 
support system had been instrumental in improving the grades of the at-risk students and 
participants were succeeding in the work-study program. However, the graduation rate among 
the students had not shown much increase. The participants in these meetings discussed the 
continuing problem of the low graduation rate among the high-risk students and possible 
solutions to the problem. 

The evaluation information and the input from the school, the community, the social service 
agencies, parents, and students indicated that a weakness in the current Dropout Prevention 
Program was the lack of interactive personal counseling between school staff and those students 
identified as potential dropouts. All participating groups and the school administration 
recommended that the program levels of academic support and work-study be maintained and 
an Interactive Counseling Service be initiated for the ninth grade at-risk students in September 
1986. They believed that this configuration would turn the dropout rate around by June 1990. 
The school-community-business partnership members would continue their active support and 
participation in the program and work cooperatively with the counselors. Representatives of the 
social service agencies would work closely with the counselors to provide high-risk students 
access to all appropriate resource services for which they or their families were eligible. 

The Sequoia superintendent and the district administrators developed a plan to expand the 
current dropout prevention program in the fall of 1986 to include an intensive one-to-one 
Interactive Counseling Service component Inservice in mentoring skills, interpersonal commu- 
nication skills and the knowledge base related to the factors that cause students to drop out of 
school were scheduled for the counseling staff. The counseling service component was expected 
to develop abonding between student and counselor and ultimately a strong connectionbetween 
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the student and the educational system. The counseling service included regular counselor 
and parent interaction to build home support for students to stay in school and graduate. 

The evaluation procedures already in place for the community involvement, the work- 
study program, and academic support services components would be continued. The new 
Interactive Counseling Service component to be introduced into the program in the fall of 
1986 would be the primary target of evaluation for the next four school terms. 

The overall goal of the Sequoia Dropout Prevention Program and the objectives and 
enabling activities of the Interactive Counseling Service component are given below* 

THE DROPOUT PREVENTION PROGRAM GOAL 

The graduation rate for at-risk students in the Sequoia United School District will be 
substantially increased by 1990. 

INTERACTIVE COU NSELING SERVICE COMPONENT 

Objective: 90% of the ninth grade at-risk students entering the program in September 1986 will 
graduate from high school 

FN APTrING ELEMENTS FOR THE COUNSELING SERVICE COMPONENT 

• Ninth- grade at-risk students will be identified early in the first term of the 1986-87 school year 
and assigned to counselors by October 15/ 1986. 

• One-to-one weekly counseling services for the students will begin the second term of the 
86-87 school year and continue in 10th, 11th and 12th grades. 

• By the 10th grade a mentor relationship will have been developed between counselor and 
student. 

• Students will be placed in the work-study program in 11th and 12th grades. 

• Counselors will meet with students and parents in August 1989 to discuss 12th grade goals. 

• Counselors will meet with work-study employers in September 1986 to discuss students' 
futures in labor markets and educational needs beyond high school. 

• Counselors and students will work out the students' strategies for success in their selected 
occupation including: employment applications, interviews, application for postsecondary 
education, and application for financial aid for postsecondaiy education. 



The examples and illustrations used in this handbook will be based on the objectives and the 
elements defined for the Interactive Counseling Service component of the program. The same 
procedures could be used to evaluate the total educational program, the school climate, and other 
community environmental factors. Evaluations the interaction among program components, school 
climate, and selected community environmental factors is also possible. 
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PHASE ONE: IDENTIFYING EVALUATION ELEMENTS 



The first step in evaluation is to identify the program components and school environmental factors 
to be evaluated. Valid sources to use in selecting the targets for evaluation are the program description 
or proposal, interviews with program developers and supporting agencies, and the stated goals and 
objectives of the program. The project director or the designated staff responsible for evaluation (we will 
call this designee the evaluation coordinator) reviews the piogram proposal, program components, 
proposed program materials, and activities then makes a list of elements to be evaluated. 

The evaluation coordinator should set up a meeting with representatives of advisory groups, 
administrators, community groups, private sector employers, social service agencies, and other groups 
directly involved with the program to determine the program components and elements and school 
environmental factors to be considered in the evaluation process. This meeting is most effective when the 
evaluation coordinator has prepared a preliminary list of elements to be evaluated for use by the 
participants as a starting point. 

If it is not possible to schedule a meeting with the vested interest groups, the evaluation coordinator 
should mail the list of elements to be evaluated to representatives of the groups and ask for additions to 
the list and their approval of the final set of elements to be evaluated. When a mailing is used with the vested 
group representatives, they should also be asked to submit a list of questions they want the evaluation to 
answer. The evaluation coordinator will use these questions in the second ph ase of the evaluation process. 

The checklist below summarizes the steps used to identify elements to be evaluated. The checklist 
can be used as a guideline in planning the activities needed to identify the program elements to be 
evaluated. 



CHECKLIST: IDENTIFICATION OF ELEMENTS TO BE EVALUATED 



1. REVIEW PROGRAM DESCRIPTION, PROPOSAL AND MATERIALS. 

2. ANALYZE OVERALL PROGRAM GOALS AND THE OBJECTIVES FOR EACH 
COMPONENT OF THE PROGRAM. 



3. PREPARE A LIST OF THE ELEMENTS WITHIN EACH PROGRAM COMPONENT 
ABOUT WHICH INFORMATION IS NEEDED. 



4. SUBMIT PRELIMINARY UST OF ELEMENTS TO BE EVALUATED TO VESTED 
INTEREST GROUPS FOR REACTION, EXPANSION AND APPROVAL. 

5. PREPARE FINAL LIST OF ELEMENTS TO BE EVALUATED. 



6. HAVE FINAL UST OF ELEMENTS APPROVED BY ADMINISTRATION AND BY 
VESTED INTEREST GROUPS INVOLVED WITH THE PROGRAM. 

7. PREPARE THE SUMMARY OF ELEMENTS TO BE EVALUATED FORM. 



9 

ERIC 



15 



7 



The Evaluation Handbook 



Nr e: The following example illustrates the use of the checklist. 



I HI M'yijl \ l Mil I) SCHOOL DISTRIC T DROl'OLT I'RI \ I \ I ION I'ROUtWI 



1 nFNTTFIC ATTON OF ELFMFNTS TO BE EVALUATED 

The evaluation coordinator for the Sequoia Dropout Prevention Program reviewed the 
program descriptions in the original proposal and interviewed the district team that developed the 
program. When the coordinator completed a summary of these documents and interviews, it 
became apparent that the focus of the evaluation process from 1986 to 1990 should be the 
Interactive Counseling Service Component An evaluation plan for the counseling service 
component would have to be completed and ready for use by September 1986. 

The evaluation coordinator analyzed the Interactive Counseling Service program description 
and the program materials and interviewed the program administrators and developers. On the 
basis of this analysis, the coordinator prepared a preliminary list of elements within the Interactive 
Counseling Services Component to be evaluated. The list of Interactive Counseling Service 
elements to be evaluated included the following: 

1. Identification of 9th grade students in fall of 1986 

2. One-to-one counseling services provided in 9th, 10th, 11th, and 12th grades 

3. Mentor relationship between counselors and students in 10th grade 

4. Counselor role during work-study student experiences in 11th grade 

5. Changes in student attitudes toward school 

6. Changes in student self-concept 

7. Changes in parent attitude toward the school system 

8. Changes in academic progress of the students 

9. Increase in the involvement of the business community in the programs 

10. Effectiveness of the counselor services in having 12th grade 
students determine their goals 

11. Changes in teaching staff attitudes of their responsibilities for keeping the 
high-risk students in school 

12. Changes in the high-risk student graduation rate 

The evaluation coordinator was not able to schedule a special meeting of representatives of 
the sponsoring and participating groups to review and expand this list Hence, the list was mailed 
to each of the groups to consider at their May 1986 meeting. The coordinator received their returns 
early in June and found there was conscensus that elements 1, 2, 3, 4, 5, 6, 7, 10, and 12 should be 
the primary foci of evaluation. The dropout rate of the students in the program was added to the 
original list 

The coordinator reviewed the final list of elements to decide when each one had to be 
evaluated and whether it was process or outcome evaluation. The coordinator then completed a 
"Summaiy of the Elements to Be Evaluated Form" for the Interactive Counselor Service Compo- 
nent as presented in Exhibit 2. 
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EXHIBIT2: SUMMARY OF ELEMENTS TO BE EVALUATED FOR INTERACT! VE 








ELEMENTS 

1 \J t V ALUM 1 C 


WHEN TO 

CWAI 1 IATC 


PROCESS OUTCOME 

PV/AI NATION FV/AI IIATinKI 
CVMLUM 1 IvIM CVMLUM 1 VJvi 


1. 1986 9th grade student 
identification 


October 86 


X X 


2. Counseling services 
provided in 9th, 10th, 
11th and 12th grades 


May 87 
May 88 
May 89 
May 90 


X X 


3. Counselor-student 
mentor relationship 
in 10th grade 


May 88 


X X 


4. Counselor role in 
student work-study 
experiences in 
11th grade 


May 89 


X X 


5. Change in student 
attitude!: ward 
school 


September 86 
May 87 
May 88 
May 89 


X 


6. Change in student 
self-concept 


May 87 
May 88 
May 89 
May 90 


X 


7. Change in parent 
attitude toward the 
school system 


May 88 
May 90 


X 


8. Change in the dropout 
rate among students in 
ine program 


May 88 
May 89 
May 5ru 


y 


9. Change in the district 
dropout rate 


June 90 


X 


10. Change in at-risk 
student posthigh 
school goals 


May 90 


X 
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PHASE TWO: GENERATING EVALUATION QUESTIONS 



Evaluation questions set the limits for the evaluation process by defining the information needed by 
the decision-making groups involved with the program. Never collect data that is not needed— don't 
generate answers to questions that no one wants answered. 

Evaluation questions should address the program's operation, the school environmental factors that 
interact with it, and its impact upon the students. (Again, a reminder that this text will only address the 
Interactive Counseling Service Component of the dropout prevention program.) An evaluation process 
that is focused upon a broad range of questions will provide the program director with information about 
the efficiency and effectiveness of the program. Without the information about the program's operation, 
it would be difficult to attribute the changes in students and environment to the program. 

Theevaluation coordinatoranalyzesthecompleted "Summary of ElementstoBe Evaluated Form" and 
produces evaluation questions for each element in the form. This first list of evaluation questions is shared 
with the program director, the administration, and vested interest groups either through meetings or by 
written correspondence. The participating persons and groups are asked to add questions to the original 
list, and all responses are analyzed for overlap and relevance. The evaluation coordinator then prepares 
a final set of evaluation questions and submits it to the program director for approval. 

The steps used in producing evaluation questions for a program are summarized in the following 
checklist. The checklist can be used as ^ guide for the generation of evaluation questions. 



CHECKLIST: STEPS IN DEVELOPING EVALUATION QUESTIONS 



1. REVIEW AND ANALYZE THE PROGRAM COMPONENT DESCRIPTION, THE LIST OF 
ELEMENTS TO BE EVALUATED, AND THE OVERALL PROGRAM GOALS AND 
OBJECTIVES. 

2. WRITE THE QUESTIONS THAT SHOULD BE ANSWERED ABOUT EACH OF THE 
PROGRAM ELEMENTS AND GOALS AND OBJECTIVES. 



3. SUBMIT THE UST OF EVALUATION QUESTIONS TO DECISIONMAKERS AND 
VESTED INTEREST GROUPS FOR REACTION, EXPANSION, AND APPROVAL. 

4. COMPLETE THE QUESTION COLUMN OF THE EVALUATION PLAN FORM 
(EXHIBIT 3). 
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I HI ' SI.QLOIA LMTID SCHOOL. DISTRICT DROPOUT PRI VI VIION PROCRAM 



The evaluation coordinator for the Sequoia Dropout Prevention Program studied the list of 
elements to be evaluated and the program goals and objectives for the counseling services. The 
coordinator found that data were needed to show that the services described in the program 
proposal had been implemented as planned and were operating efficiently. The coordinator also 
knew that data were needed to show the impact of each major counseling service on the at-risk 
students in the program and the overall impact of the counseling service on the district's dropout 
rate. Using these two overriding information needs and the "Summary of Elements to Be 
Evaluated/ the coordinator developed a set of questions to provide a focus for the evaluation. 

A meeting of representatives of the administration and the vested interested groups was held 
to review the evaluation questions produced by the evaluation coordinator and to develop 
additional questions. At this meeting some of the original evaluation questions were refined, 
others were discarded as unnecessary, and several additional questions were added to the list 
The participants in the meeting submitted the list of questions to their agencies for approval. The 
evaluation coordinator entered the questions on the "Evaluation Plan Form for Interactive 
Counseling Services" as shown on Exhibit 3. 
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itlon Questions 



Objective or Experimental Data Sources 
EPement Design 



Data Analyses 



Data Collection 
Schedule 



Report Schedule 



1FSB EVALUATION 



at-risk students effectively 
fied and assigned to 
selors by 10/15/86? 



Element 1 



le percentage of the total 
•grade class identified and 
ned to counselors equal or 
ad the average dropout rate 
e district over the last ten 
>? 



Element 1 



ill at-risk students in the 
ram meet with their 
selors once each week 
\g the ninth, tenth, eleventh 
twelfth grades? 



Element 2, 3, 4 



aarents participate in at 
t ten percent of the weekly 
iseling sessions? 



Element 2, 3, 
4,7 



O 1 



EXHIBn 3: EVALUATION PL AN FOR INTERACTIVE COUNSELING SERVICES 



Evaluation Questions Ob|active or Experimental Data Sources Data Analyses Data Collection Report Schedule 

Element Design Schedule 



OUTCOME EVALUATION 



Did a mentor relationship Element 3 

develop between counselor and 
student by the end of the tenth 
grade? 



Did all students have access to Element 4 
a work-study experience in the 
eleventh grade? 



Did all students enter the twelfth Element 5,10 
grade with broad goals for post- 
high school? 



Did 90 percent of the 1990 class 
develop personal plans for post- 
high school requiring a high 
school diploma? 



Objective 1 



Did stu" jnts follow their plans or Objective 1 
^dapt them after graduation? 



Did student attitudes toward 
school reflect positive changes 
after they participated in the 
program? 



Element 5, 6 



Did the district dropout rate 
show a significant decline 
between 1986 and 1990? 



Program Goal 
Element 8, 9 
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PHASE THREE: SELECTING A DESIGN, COLLECTING AND 
ANALYZING DATA 

Evaluation designs are directly related to the evaluation questions and the information needed to 
answer these questions. The evaluation design, in turn, determines the data collection schedule and the 
range of procedures that maybe used to analyze the data collect id. The data summarization and reporting 
format are influenced bythenatureof the audiencefor the evaluation report. For many evaluation questions, 
the decisionmakers and vested groups are interested in head counts and percentages; for others they may 
wish to use averages or proportions to compare the program participants with district, state or national 
norms. All of these factors must be considered when choosing an evaluation design and data analysis 
procedures. 

Th is section provides a brief description of evaluation designs and data analysis procedures which could 
be used with these designs. 



EVALUATION DESIGNS 

An evaluation design specifies the data to be collected, how it will be collected, when it will be collected, 
and from whom it will be collected. An appropriate evaluation design will generate the highest quality data 
possible within the constraints imposed by the real-world environment of a program. 



Evaluation designs that could be used for dropout prevention programs are: 



ONE-GROUP IN THIS DESIGN, DATA COLLECTED AT THE END OF A 

POSTTEST ONLY PROGRAM ARE COMPARED WITH A CRITERIA SPECIFIED IN 

A GOAL OR AN OBJECTIVE. IT IS NOT A STRONG DESIGN BUT 
IN THE REAL WORLD OF AN EDUCATIONAL SYSTEM IT WILL 
OFTEN PROVIDE THE INFORMATION NEEDED BY THE 
DECISIONMAKERS AND VESTED INTEREST GROUPS. 



ONE-GROUP THIS DESIGN IS STRONGER THAN THE ONE-GROUP POSTTEST 

PRETEST-POSTEST ONLY BECAUSE DATA COLLECTED AT THE BEGINNING AND AT 

THE END OF A PROGRAM ARE COMPARED TO SHOW CHANGES 

IN PARTICIPANTS. 



ONE-GROUP THE ONE-GROUP TIME SERIES DESIGN REQUIRES DATA 

TIME SERIES COLLECTION AT MULTIPLE TIMES DURING A PROGRAM. BY 

COMPARING INFORMATION COLLECTED OVER TIME, CHANGES 
IN PARTICIPANTS CAN BE COMPARED TO SHOW IMPROVEMENT 
DURING A PROGRAM. 
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PRETEST-POSTTfcST THIS TWO-GROUP DESIGN REQUIRES RANDOM ASSIGNMENT 

WITH CONTROL GROUP OF THE PARTICIPANTS TO A PROGRAM GROUP AND TO A 

CONTROL GROUP THAT WILL NOT RECEIVE THE PROGRAM. 
PRETEST AND POSTTEST DATA ARE COLLECTED ON BOTH 
GROUPS AND THEIR POSTTEST GAINS ARE COMPARED. 



POSTTEST ONLY WITH RANDOM SAMPLING AND RANDOM ASSIGNMENT OF 
CONTROL GROUP PARTICIPANTS TO PROGRAM AND CONTROL GROUPS 

ARE REQUIRED WHEN USING THIS DESIGN. POSTTEST 
DATA ARE COLLECTED AND THE SCORES OF THE TWO 
GROUPS ARE COMPARED. 



NONEQUIVALENT THIS DESIGN USES AN INTACT GROUP THAT APPEARS 

CONTROL GROUP TO BE SIMILAR TO THE PROGRAM GROUP. AN V DATA 

COLLECTED FROM THE PROGRAM GROUP ARt ALSO 
COLLECTED FROM THE CONTROL GROUP AND 
COMPARISONS ARE MADE. FREQUENTLY DATA FROM 
SCHOOL SY r "*EM RECORDS FOR THE CONTROL GROUP 
ARE COMPAhED WITH DATA COLLECTED ON THE 
PROGRAM GROUP. THIS DESIGN DOES NOT REQUIRE 
RANDOM SAMPUNG AND ASSIGNMENT. 



The control group designs are stronger than the one-group designs and give more freedom to attribute 
changes in participants to the program, but most of them require random sampling and random assignment 
to groups. The use of randomization to form experimental and control groups means that program services 
are withheld from the at-risk students in the control group. For most dropout prevention programs, this is 
an unacceptable condition. 

Any of the one-group designs or the Nonequivalent Control Group design would be appropriate for use 
in evaluating dropout prevention programs. Although One-Group Posttest Only design is the weakest of 
the designs, it is an efficient design for process or implementation evaluation. 

The One-Group Pretest-Posttest design, the One-Group Time Series design and the Nonequivalent 
Control Group design should be used for outcome evaluation. When assessing differences in students due 
to program participation, the One-Group Pretest-Pottest design should be used. To assess the ongoing 
impact of the program, the One-Group Time Series would be the design of choice. To compare the effect 
of the program on overall dropout rate, the Nonequivalent Control Group design is appropriate using the 
district's preprogram five year average dropout rate of similar students as the data for the control group. 

Detailed descriptions oi the designs described here can be found in the evaluation literature. More 
complex and sophisticated designs can be found in both the evaluation literature and in resource books 
on research designs. A source on data analyses and statistical procedures helpful to evaluators is 
Research and Education (Best of Kohn, 1989). When a complex study with true experimental design is 
planned, the program coordinator should seek expert advice from evaluation consultants. 



DATA COLLECTION AND ANALYSIS 



The data collection and analysis procedures will vary with the evaluation questions and with the 
evaluation design selected for each question. The evaluation design dictates the schedule for data 
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collection while the evaluation questions determine the type of data collection instruments that will be 
needed and the type of data analysis required. Questions addressing student academic achievement can 
be answered using standardized achievement tests or curriculum imbedded tests. If questions are asked 
about changes in student attitudes or self concept, then valid and iliable instruments must be located or 
constructed. 

When a two-group design with randomization and with pre- and posttesting are used, data analyses 
such as T-Tests and Analysis of Variance can be used to determine significant posttest differences between 
the control group and the program group. If this is the type of design selected then the program director 
should have evaluation experts do the statistical analyses needed. Personal computer statistical packages 
are available that can handle this level of sophistication in data analysis. 

The designs most likely to be successfully carried out in the "real world" educational system are any of 
the one-groupdesigns and the Nonequivalent Control Group design. The choice of a design depends upon 
the evaluation questions and on the human, financial and time resources available for data collection and 
analysis. 

The One-Group Posttest Only design can be used to compare end of the program participant status with 
criteria given in program goals and objectives. Data can be analyzed using descriptive statistics such as 
the percentage of students achieving the criteria stated in each objective. The magnitude of this percentage 
is easily interpreted for its practical significance. 

The One-Group Pretest- Posttest design requires data collection at the start of the program and at the 
end of the program. The participants' scores on the pretest and posttest are compared to determine the 
impact of the program. Descriptive statistics can be jsed to show the pre- and the posttest average scores 
and the percentage of students whose posttes* scores showed the expected level of increase over their 
pretest scores. Statistics such as the Correlated t-Test can be used to show statistical significance of the 
difference between the pretest mean and the posttest mean. Chi Square can be used to compare actual 
gain with expected gains. 

The Nonequivalent Control Groupdesign can be used to compare the programgroup data with baseline 
data on past classes in the educational system. For dropout prevention programs, this could mean 
comparing the dropout rate of the program participants with the average dropout rate of the school over the 
preceding five year period. The magnitude of the difference between the proportion of dropouts in the two 
groups will be apparent when tney are viewed. Use of the Test for Differences in Proportion will provide the 
statistical significance of the difference oetween thd dropout rate of the two groups. 

The One-Group Time Series design requires data collection on the program group at several points in 
time during the program's operation. If a program is operating for one school yea.', data could be collected 
at the start of the program, at the midpoint of the program, and at the end of the program. A program 
operating over several school years should collect data at the beginning and end of each year. Several 
approaches to data analysis may be used— the average scores from each testing could be plotted on line 
graphs to show upward trends during the program, bar graphs could show changes from term to term or 
data could be displayed in clearly marked tables with narrative interpretations. Exhibit 4 illustrates 
evaluation design types and related data analyses. 
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■'% EXHIBIT 4: EVALUATION DESIGN TYPES'A^R 


DESIGN TYPE 




DATA ANALYSES 


DESIGN 1: ONE-GROUP POSTTEST ONLY 




FREQUENCY COUNTS, PERCENTAGES & 
PROPORTIONS, MEAN SCORES, ETC. 


DESIGN 2: ONE-GROUP PRETEST-POSTTEST 




DIFFERENCES IN PROPORTION, CORRELATED 
T-TEST OF DIFFERENCE IN MEANS, 
CHI SQUARE 


DESIGN 3: ONE-GROUP TIME SERIES 




FREQUENCY COUNTS, PERCENTAGES & 
PROPORTION SHOWN FOR TEST SCORES 
OVER MULTIPLE TESTING DATES, 
CORRELATED T- TEST OF DIFFERENCE IN 
MEANS BETWEcN FIRST AND LAST TESTING 


DESIGN 4: PRETEST-POSTTEST WITH 
CONTROL GROUP 




ANALYSES OF VARIANCE, 
INDEPENDENT T-TEST 


DESIGN 5: POSTTEST ONLY WITH CONTROL GROUP 


DIFFERENCES IN PROPORTION 


DESIGN 6: NONEQUIVALENT CONTROL GROUP 




DIFFERENCES IN PROPORTION, 
FREQUENCY COUNTS 


DESIGN 

REQUIREMENTS 1 2 


3 


DESIGN TYPES 
4 5 6 


ONE GROUP Y Y 


Y 




TWO GROUPS 




Y Y Y 


PRETEST Y 




Y 


POSTTEST Y Y 




Y Y Y 


SERIES OF TEST 


Y 




RANDOMIZATION 




Y Y 


Y=Yes, indicating design types applicable to specific design requirements 



The steps to follow in selecting an appropriate evaluationdesign are summarized in thefollowing check- 
list. The checklist can be used by an evaluation coordinator as a guideline in identifying the design to be 
used in an evaluation study. 



n "1 
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CHECKLIST: STEPS IN EVALUATION DESIGN SELECTION 



1 . FOR EACH EVALUATION QUESTION ANSWER THE FOLLOWING ITEMS. 

. HOW MANY GROUPS WILL BE IN THE EVALUATION STUDY? 

. WHAT KIND OF DATA WILL PROVIDE THE ANSWER TO EACH OF THE 
EVALUATION QUESTIONS? 

• WHEN WILL DATA BE COLLECTED? POSTTEST? PRETEST AND POSTTEST? 
SERIES OF TESTS OVER TIME? 

* WILL RANDOMIZATION BE USED? 

2. USE YOUR ANSWERS TO THE ITEMS ABOVE TO SELECT AN EVALUATION DESIGN AND 
DATA ANALYSIS PROCEDURES AND NOTE ON EXHIBIT 5. 
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SELECTION OF THE EVALUATION DESIGN AND DATA ANALYSIS PROCEDURES 

The evaluation coordinator for the Sequoia District Dropout Prevention Program studied the 
evaluation questions to select an appropriate evaluation design for each one. The coordinator used 
the checklist given above and then compared the information about each evaluation question with 
the chart given in the checklist 

The coordinator found that for some of the process evaluation questions there was no need for 
a design. It was decided that an effort would be made to collect program data and simply report the 
results in the form of simple percentages or with anecdotal notes to describe the program activities. 

The coordinator found that with the exception of the evaluation question asking for a compari- 
son of the 1990 dropout rate of the program at-risk students with the districf s average dropout rate 
for the previous five years, data needed to be collected on only one group. The coordinator selected 
an evaluation design for each evaluation question and entered the designs in the evaluation plan as 
illustrated in Exhibit 5 which follows. 
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Evaluation Questions Objective or Experimental Data Sources Data Analyses Data Collection Report Schedule 

Element Design Schedule 



PROCESS EVALUATION 



Were at-risk students effectively 
identified and assigned to 
counselors by 10/15/86? 



Element 1 No design 

needed 



Did the percentage of the total 
ninth-prude class identified and 
assigned to counselors equal or 
exceed the average dropout rate 
for the district over the last ten 
years? 



Element 1 Nonequivalent 
control group 



Did all at-risk student?: in the Element 2, 3, 4 One-group 
program meet with their time series 

counselors once each week 
during the ninth, tenth, eleventh 
and twelfth grades? 



Did parents participate in at Element 2, 3, One-group 

least ten percent of the weekly 4, 7 time series 

counseling session? 



0 Cj 



3: 



EXHIBIT 5: EVALUATION PUN FOR INTERACTIVE COUNSELING SERVICES 




Evaluation Question 


Objective or Experimental Data Sources Data Analyses 
Element Design 


Data Collection 
Schedule 


Report Schedule 



OUTCOME EVALUATION 



Did a mentor relationship 
develop between counselor and 
student by the end of the tenth 
grade? 



Element 3 One-group 
posttest only 



Did all students have access to 
a work-study experience in the 
eleventh grade? 



Did all students enter the twelfth 
grade with broad goals for post- 
high school? 



Did 90 percent of the 1 990 class 
develop personal plans for post- 
high school requiring a high 
school diploma? 



Did students follow their plans or 
adapt thorn after graduation? 



Did student attitudes toward 
school reflect positive changes 
after they participated in the 
program? 



Element 4 One-group 
posttest only 



Element 5,10 One-group 
posttest only 



Objective 1 One-group 
posttest only 



Objective 1 One-group 
posttest only 



Elements, 6 One-group 
time series 



31 



Did the district dropout rate Program Goal Nonequivalent 

show a significant decline Element 8, 9 control group 

between 1986 and 1990? ')? 
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PHASE FOUR: INSTRUMENTATION 

The 6 r ,\a collection instruments needed for an evaluation are determined by the kind of information 
needed to answer each of the evaluation questions. It may be necessary to have a separate instrument 
for each question, but usually items addressing several questions can be incorporated into one instrument. 
The items are clustered into subtests and data from each subtest are analyzed separately for individual 
questions. Some questions can be answered using information from existing student and program records. 

The instruments used for data collection fall into several broad categories. Three categories are used 
for illustration. 

MEASURES OF ACADEMIC ACHIEVEMENT 

• NORM REFERENCED STANDARDIZED TEST 

• CRITERION REFERENCED TESTS 

• CURRICULUM EMBEDDED TESTS 

• TEACHER MADE ACHIEVEMENT TESTS 
MEASURES OF PROGRAM GOALS AND OBJECTIVES 

• STUDENT OPINIONAIRES 

• ATTITUDE MEASURES 

• STUDENT ACTIVITY LOGS 

• STAFF DAILY LOGS 
. PARENT SURVEYS 

• COMMUNITY SURVEYS 

• INSERVICE EVALUATION QUESTIONNAIRES 

MEASURES OF SCHOOL CLIMATE AND COMMUNITY E NVIRONMENTAL FACTORS 

• COMMUNITY/BUSINESS SURVEYS 

• RECORDS OF COOPERATIVE ACTIVITIES BETWEEN SOCIAL SERVICE AGENCIES, 
ADMINISTRATION AND STAFF MEMBERS, PROGRAM STAFF, AND BUSINESS 
REPRESENTATIVES 

• RECORDS OF BUSINESS/COMMUNITY PARTICIPATION 
. ATTITUDE MEASURES OF COMMUNITY GROUPS 



O 
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• ASSESSMENT OF THE SCHOOL CLIMATE 

• PUBLIC OPINION POLLS 

. RECORDS OF VOLUNTEER ASSISTANTS 

. RECORDS OF INTERACTION AMONG PROGRAM STAFF AND COMMUNITY ORGANIZATIONS 

Instruments already developed with proven validity and reliability standards would be ideal but may not 
always be available, therefore, instruments to measure gains in program goals and objectives would need 
to be constructed by the evaluator and program staff. Items in such instruments should be clearly stated 
and each Item should address one andonly one concept. Item response format is determined by the nature 
of the item. Response choices may be as simple as yes-no or checking each appropriate response. 
Response format can also be a 4- or 5-point scale with anchor points such as agree-disagree. Scaled 
responses may also have a specific descriptive statement for each point on the scale determined by the 
nature of the item. 

All newly constructed instruments should be field tested with a group representative of the persons who 
will be responding to it In the evaluation study. The field test identifies confusing or ambiguous items and 
items that fail to discriminate among test takers. Ambiguous items should be rewritten. Nondiscriminating 
items should be discarded or replaced unless you expect the same response from all persons. 

The steps to follow in selecting or constructing data collection instruments are summarized in the 
following checklist. It can be used by an evaluation coordinator as a guide in developing the instrumentation 
needed for evaluation. 



CHECKLIST: INSTRUMENTATION 



1. FOR EACH EVALUATION QUESTION, DETERMINE FROM WHOM YOU WILL NEED DATA: 
. STUDENTS 
. STAFF 
. PARENTS 

• BUSINESS LEADERS 

• WORK EXPfc flIENCE COORDINATORS 

• SOCIAL SERViCE AGENCIES 
. COMMUNITY GROUPS 

. GENERAL PUBUC 
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2. FOR EACH EVALUATION QUESTION LIST THE MAJOR CATEGORIES OF DATA 
NEEDED: 

• ACADEMIC ACHIEVEMENT DATA 

• CDUNT OF STUDENTS MEETING DEFINED CRITERION LEVELS 

• HOURS OF SPECIAL SERVICES RECEIVED 

• STUDENT OPINION DATA 

• DATA ON ATTITUDINAL CHANGES 

• HOURS OF WORK EXPERIENCE 

• QUALITY OF WORK PERFORMANCE 

• PARTICIPATION LEVEL OF COMMUNITY, BUSINESSES AND SOCIAL AGENCIES 

3. IDENTIFY AND OBTAIN INSTRUMENTS AVAILABLE TO COLLECT THE NEEDED DATA 

4. IDENTIFY INSTRUMENTS THAT MUST BE DEVELOPED 

5. SELECT STAFF TO WRITE ITEMS FOR THE QUESTIONNAIRES 

6. CONSTRUCT INSTRUMENTS: 

• FIELD TEST INSTRUMENTS 

• REVISE INSTRUMENTS ON THE BASIS OF FIELD TEST DATA 

• PRODUCE THE INSTRUMENTS TO USE IN DATA COLLECTION 



The following example illustrates the use of the instrumentation checklist. 
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SELECTING DATA COLLECT ION INSTRUMENTS 



The evaluation manager for the Sequoia Dropout Prevention program used the checklist for 
instrumentation to identify the instruments needed to answer the evaluation questions in the 
program evaluation plan. The instruments needed included a Student Survey, a Parent Survey, 
a Counselor Log, and a form for use in gathering existing data from school records. All of these 
instruments would be constructed and f ieldtested by program staff. The manager then listed the 
instruments in the evaluation plan as shown in Exhibit 6. 
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Evaluation Questions 



Objective or 
Element 



Experimental 
Design 



Data Sources 



Data Analyses 



Data Collection 
Schedule 



Report Schedule 



PROCESS EVALUATION 



Were at-risk students effectively Element 1 
identified and assigned to 
counselors by 10/15/86? 



No design Counselor records 
needed Student files 



Did the percentage of the total Element 1 Nonequivalent student records 

ninth-grade class identified and control group 

assigned to counselors equal or 

exceed the average dropout rate 

tor the district over the last ten 

years? 



Did all at-risk students in the 
program meet with their 
counselors once each week 
during the ninth, tenth, eleventh 
and twelfth grades? 



Element 2, 3, 4 



One-group 
time series 



Counselor logs 
Student surveys 



Did parents participate in at Element 2, 3, One-group 

least ten percent of the weekly 4, 7 time series 

counseling sessions? 



Parent survey 
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EXHIBIT 6: EVALUATION PLAN FOR INTERACTIVE COUNSELING SERVICES 



Evaluation Questions 



Objective or Experimental Data Sources 
Element Design 



Data Analyses 



Data Collection 
Schedule 



Report Schedule 



QUTCQME EVALUATION 



Did a mentor relationship 
develop between counselor and 
student by the end of the tenth 

grade? 



Element 3 One-group Student survey 

posttestonly Parent survey 



Did all students have access to 
a work-study experience in the 
eleventh grade? 



Element 4 



One-group 
posttest only 



Counselor logs 
Student survey 



Did all students enter the twelfth 
grace with bruad goals for post- 
high school? 



Element 5,10 



One-group 
posttest only 



Counselor logs 
Student survey 



Did 90 percent of the 1990 class 
develop personal plans for post- 
high school requiring a high 
school diploma? 



Objective 1 



One-group 
posttest only 



Counselor logs 
Student survey 
Student files 



Did students follow their plans or Objective 1 One-group 
adapt them after graduation? posttest only 



Follow-up student survey 



Did stucent attitudes toward 
scnool reflect positive changes 
after ihey participated in the 
program? 



Element 5, 6 



One-group 
time series 



Student survey 
Parent survey 



Did the district dropout rate 
show a significant decline 
between 1986 and 1990? 



Proyram Goal Nonequivalent School record* 
Element 8, 9 control group 
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PHASE FIVE: COLLECTING, SUMMARIZING AND 
INTERPRETING DATA 

The success of any program evaluation depends upon the efficient collection, analysis, and interpre- 
tation of evaluation data. The evaluation coordinator and the program director must develop and execute 
a plan for the collection, summary, and interpretation of program data. Data collection times and sites must 
be set up, staff must be trained to collect the data, responsibility for data summarization, and interpretation 
must be assigned. 



DATA COLLECTION 

The data collection time schedule is determined by the evaluation design and the data report needs of 
the decisionmakers. When a design requires pretest data collection before participants receive program 
services, then the first data collection instrument must be administered as soon as possible after participants 
are enrolled in a program. Posttests are administered as close as possible to the ending date of the program. 
When repeated administration of an instrument to the same participants during the program is planned, 
these test dates should be scheduled at planned intervals and the same sequence maintained every year. 

If the design requires that data be abstracted from existing school records or from staff logs, then 
standardized forms should be used to collect this information. 

It is important that all data collection instruments are obtained at least two months before the start of 
the program. When instruments have to be constructed by the evaluation staff, the target date for 
completion should be no later than th'ee months before the start of the program to allow time for field testing 
before printing. At least one month before the program begins, testing dates must be set and test 
administrators trained. 



DATA SUMMARY AND INTERPRETATION 

The procedures used to analyze the evaluation data are decided upon when the evaluation design is 
selected. The data should be submitted for analysis as soon as possible after it is collected. Each statistical 
approach to data analyses produces summary data that can be displayed in tables, graphs, and charts. The 
summary data displays will clearly indicate the effect of the program upon participants arid narrative 
interpretations should reflect only the facts supported by the data. 

The interpretation of the data summaries is the responsibility of the evaluator coordinator. One 
approach to data interpretation is to summarize the concepts measured by the instruments and relate them 
to th<° program activities and objectives. Data interpretation is always a narrative statement backed up by 
the summary charts, tables, and figures. It is important that the coordinator does not go beyond the data 
presented when writing interpretations— you can't say that a program had an effect upon a concept or 
behavior that wasn't measured during data collection. 

The steps to follow for data collection, data summary, and interpretation are summarized inthe following 
checklist. It can be used as a guideline for program directors and evaluators in planning for these 
procedures. 
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CHECKLIST: STEPS IN DATA COLLECTION, SUMMARY AND INTERPRETATION 



1. EXAMINE THE EVALUATION DESIGN AND SET DATA COLLECTION DATES: 



. PRETESTS SHOULD OCCUR A WEEK BEFORE THE PROGRAM SERVICES BEGIN. 



• POSTTESTS SHOULD OCCUR NO EARLIER THAN THE FINAL MONTH OF THE 
PROGRAM AND PREFERABLY DURING THE LAST WEEK OF PROGRAM SERVICES. 



2. SET DATES FOR EVALUATION REPORTS. 



3. ORDER OR PRINT ALL INSTRUMENTS AND DATA COLLECTION FORMS SO THAT THEY ARE 
AVAILABLE 12 WEEKS BEFORE TEST DATES. IF THERE ARE PROBLEMS WITH DELIVERY, 
THE LATEST THAT INSTRUMENTS SHOULD BE AVAILABLE IS ONE MONTH BEFORE THE 
START OF PROGRAM SERVICES. 



4. ONE MONTH BEFORE THE FIRST TEST DATE, TRAIN DATA COLLECTION STAFF IN 

ADMINISTRATION OF THE INSTRUMENTS. 



5. COMPLETE DATA COLLECTION. 



6. ANALYZE AND SUMMARIZE DATA FOR EACH EVALUATION QUESTION. 



7. PREPARE TABLES, GRAPHS AND CHARTS TO DISPLAY DATA SUMMARIES AND WRITE 
NARRATIVE DATA INTERPRETATIONS FOR EACH QUESTION. 
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DAT A COLLECTION, SUMMARIZATION AND INT E RPRETATION 

The evaluation coordinator for the Sequoia School District Dropou t Prevention Program had 
completed the evaluation plan for the program and began preparation for implementing it and 
collecting the data. After listing the instruments and data collection forms, the coordinator 
determined the appropriate data analysis method, set specific dates for each test and entered this 
information on the evaluation plan (Exhibit 7). It was apparent that counselor logs would have 
to be constructed and a data collection format was needed for use in abstracting data from school 
records. The student survey, patent survey and the student follow-up survey would have to be 
developed by the evaluator and program staff because these instruments were program specific 
in thr areas to be measured. 
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EXHIBIT 7: EVALUATION PLAN FOR INTERACTIVE COUNSEl ING SERVICES 



Evaluation Questions 



Objective or Experimental Data Sources 
Element Design 



Data Analyses 



Data Collection 
Schedule 



Report Schedule 



PROCESS EVALUATION 



Were at-risk students effectively 
identified and assigned to 
counselors by 10/15/86? 



Did the percentage of the total 
ninth-grade class identified and 
assigned to counselors equal or 
exceed the average dropout rate 
for the district over the last ten 
years? 



Did all at-risk students in the 
program meet with their 
counselors once each week 
during the ninth, tenth, eleventh 
and twelfth grades? 



Element 1 No design Counselor records 

needed Student files 



Student count 



Element 1 



Nonequivalent student records 
control group 



Element 2, 3, 4 



One-group 
time series 



Counselor logs 
Student surveys 



Comparison of 
proportions 



Comparison of totals 
and percentages jach 
year 



10/18/86 



10/18/86 



5/15/87 
5/15/88 
5/15/89 
5/15/90 



10/30/86 



10/30/86 



6/30/87 
6/30/88 
6/30/89 
6/30/90 



Did parents participate in at Element 2, 3, 

least ten percent of the weekly 4, 7 
counseling sessions? 



One-group 
time series 



Parent survey 



Comparison of totals 
and percentages each 
year 



5/15/87 
5/15/88 
5/15/89 
5/15/90 



6/30/87 
6/30/88 
6/30/89 
6/30/90 
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EXHIBIT 7: EVALUATION PLAN FOR INTERACTIVE COUNSELING SERVICES 



Evaluation Questions 



Objective or Experimental Data Sources 
Element Design 



Data Analyses 



Data Collection 
Schedule 



Report Schedule 



OUTCOME EVALUATION 



Did a mentor relationship 
develop between counselor and 
student by the end of the tenth 
grade? 



Element 3 



One-group 
posttest only 



Student survey 
Parent survey 



Item analysis 
Comparison of 
percentages 



5/15/88 



6/30/88 



Did all students have access to 
a work-study experience in the 
eleventh grade? 



Element 4 



One-group 
posttest only 



Counselor logs 
Student survey 



Student count 



5/15/89 



6/30/89 



Did all students enter the twelfth 
grade with broad goals for post- 
high school? 



Element 5,10 



One-group 
posttest only 



Counselor logs 
Student survey 



Analysis of student 
plans to identify goals 



9/20/89 



10/15/89 



Did 90 percent of the 1 990 class 
develop personal plans for post- 
high school requiring a high 
school diploma? 



Objective 1 



One-group 
posttest only 



Counselor logs 
Student survey 
Student files 



Comparison of achieved 
percentage with criteria 



5/28/90 



6/30/90 



Did students follow their plans or Objective 1 One-group 
adapt them after graduation? posttest only 



Follow-up student survey 



Comparison of 
achievement with plans 



4/20/91 



6/30/91 



Did student attitudes toward 
school reflect positive changes 
after they participated in the 
program? 



Element 5, 6 



One-group 
time series 



Student survey 
Parent survey 



Trend analysis 
comparison of mean 
scores 



9/15/86 
5/15/87 
5/15/88 
5/15/89 



6/30/87 
6/30/88 
6/30/89 
6/30/90 



Did the district dropout rate 
show a significant decline 
between 1986 and 1990? 



Program Goal 
Element 8, 9 



Nonequh/alent 
control group 



School records 



Comparison of 
proportions 



5/15/90 



6/30/90 
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A student survey was constructed that had four subsections; each section addressed an 
evaluation question. The parent survey was designed to assess three of the evaluation questions. 
A counselor log and a student data form were designed to collect the specific data from counselor 
records and from school records to answer five of the evaluation questions. All jf the instruments 
were field-tested inMay 1986 and were revised on thebasis of field-test data. All instruments were 
printed and delivered by October 1986. 

The evaluation coordinator had identified a central office person with expertise in data 
preparation, analyses, and summarization. As soon as data collection on each testing date was 
completed, the data were sent to the central office person for analysis. The data analyses and 
summaries were returned to the evaluator for review. The coordinator then prepared visual 
displays of the data and data interpretations for each of the evaluation questions. Interim 
evaluation reports were delivered to the decisionmakers and vested interest groups at the end of 
each school term, and the final evaluation report was delivered in June 1990. 

Sample data summaries and interpretations are given for two of the evaluation questions in 
the evaluation plan for the Sequoia Dropout Prevention Program. The evaluation question is 
presented followed by a description of the design and the data collection and analysis procedures. 
Visual presentation of the data summaries are shown with narrative data interpretations needed. 

A SAMPLE DATA SUMMARY AND INTERPRET *■ HON 

EVALUATION QUESTIO N 

Was there a positive change in student attitudes toward school between entry into the 
program and high school graduation? 

PROCEDURES 

The at-risk students in the Sequoia Dropout Prevention Program received intense interactive 
one-to-one counseling services each week during their four years in high school. Their attitude 
toward school was measured by items in the student survey administered at the first counseling 
session in January 1987 and then in May of each school year from 1987 to 1990. The data collected 
over the four year period are displayed in Exhibit 8. 
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EXHIBITS: ^ 



Average Score 



Test 9/86 
Dates 



5/87 



5/88 



5/89 



5/90 



Inter pretation of Student Attitu de Data 

From the data displayed in Exhibit 8, one can see that the attitude toward school of the at-risk 
students in the Dropout Prevention Program at the Sequoia High School did change from a very 
negative attitude to a highly positive one. Inl986-87, the f irstyear that the students were receiving 
the interactive counseling services there was no change in the attitude of the students. In 1986, 
the average score on the attitude-toward-school measure was 1 on a 5 point scale (5 was the highest 
possible score and 1 the lowest possible score). By the end of May 1988, the average score on the 
attitude toward school measure had moved up to 2.5. 

In May 1989, the average attitude-toward-school score of the at-risk students in the program 
had reached 3.5 and in May 1990 this average score was 4, indicating a highly positive attitude 
toward school. 
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EVALUATION QUESTION 

Was there a significant drop in the district dropout rate between 1986 and ?.990? 
PROCEDURES 

A test of difference in proportions was used to assess the change in district dropout rates 
between 1986 and 1990. The district average dropout rate for 1981-86 was used as the baseline 
proportion and the dropout rate for 1990 was compared with the baseline data. 

INTERPRETATION 

A review of Exhibit 9 shows that the dropout rate in the district decreased by 60 percent 
between 1986 and 1990. This difference had practical significance for the district in that it showed 
that the at-risk students in the program were staying in school and graduating. With this great a 
difference in proportion it was not necessary to test for statistical significance but the evaluation 
manager did run the Test for Difference in Proportions and found the difference statistically 
significant at the .05 level. This statistically significant difference supported the evidence that the 
Dropout Prevention Program was keeping more at-risk students in high school until graduation. 



EXHIBIT 9: COMPARISON OF DISTRICT FIVE YEAR AVERAGE DROPOUT AAT6 AND THE 1990 
DROPOUT RATE 



District Average 
Dropout Rate 



1990 
Dropout Rate 



Difference 



.25 



.10 



.15* 



'Significant at the .05 level 
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PHASE SIX: THE EVALUATION REPO r T 



Every evaluation requires reporting the findings to decisionmakers, vested interest groups, and the 
general public. There are strategic points during the program operation when interim evaluation reports 
are needed for operational decisions and program adjustment. The sponsoring and funding agencies will 
require reports on progress toward program goals during its operation and when the participants complete 
the program. 

The evaluator has the primary responsibility for producing all of the evaluation reports and delivering 
them to the program director. The distribution, publication and presentation of the evaluation reports are 
the program director's responsibility, but the evaluator is usually involved in the presentation of the reports. 

An evaluation report is organized around the evaluation questions and should contain the following 
sections: 



1 . a brief description of the program and its goals and objectives; 

2. the evaluation questions; 

3. for each evaluation question: 

• a description of the evaluation design; 

• a description of the data collection procedures for each target group including 
instrumentation, data analysis procedures and the rationale for their use; 

• the data summaries, interpretation and illustrative visual displays of the data; 

• summary of the findings; and 

4. an executive summary of the evaluation report that can be distributed independent 
of the total report. 

The language in an evaluation report should be clear, concise, and free of highly technical terms and 
educational jargon. The data summaries and interpretations should be easily understood by readers 
without an evaluation background. An effective evaluation report will be understood by thetarget audiences 
without additional explanations from the evaluator. 

Decisionmakers should be able to justify program changes, continuation, expansion, or termination on 
the basis of the data presented in the evaluation report. 

A checklist to use as a guide in preparing an evaluation report follows. 
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CHECKLIST: EVALUATION REPORT PREPARATION 



1. WRITE A BRIEF DESCRIPTION OF THE PROGRAM, ITS GOALS AND OBJECTIVES. 

2. UST ALL EVALUATION QUESTIONS. 

3. SORT ALL DATA SUMMARIES AND INTERPRETATIONS BY EVALUATION 
QUESTIONS. 

FOR EACH EVALUATION QUESTION, WRITE A DESCRIPTION OF THE EVALUATION 
DESIGN, DATA COLLECTION PROCEDURES, DATA ANALYSES, DATA SUMMARIES 
AND INTERPRETATION. 

5. WRrTE THE EXECUTIVE SUMMARY OF THE EVALUATION REPORT. 

6. SUBMIT A DRAFT OF THE EVALUATION REPORT TO THE PROJECT MANAGER FOR 
REACTION AND APPROVAL 

7. MAKE REVISIONS AND REFINEMENTS OF DRAFT AND SUBMIT TO PROJECT 
MANAGER FOR FINAL APPROVAL. 

8. PREPARE THE FINAL COPY OF THE EVALUATION REPORT AND TURN IT OVER TO 
THE PROJECT MANAGER FOR PUBLICATION AND DISTRIBUTION. 



SUMMARY OF THE EVALUATION PROCESS 

This section presented the six steps in the evaluation process and a checklist and illustrative examples 
for each step. The examples focused upon only one component of a dropout prevention program and its 
impact upon students. When evaluating a program in a real-world situation, all components would be 
assessed and their impact upon parents, teachers, overall institutional attitude toward the high-risk student, 
and community involvement would be measured. 

The next section will provide some direction in assessing overall program outcomes that may not have 
direct student impact, but do influence program direction. 



EVALUATING OVERALL IMPACT OF THE DROPOUT PREVENTION PROGRAM 

The preceding sections presented the evaluation process in detail. The same process and the checklist 
for each of the six evaluation phases should be used to measure the operation and impact of all components 
(e.g., community involvement, parental interest.etc.) of a dropout prevention program. 
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The degree of community involvement and its impact upon the public acceptance of a responsibility 
and a role in preventing high-risk student dropouts should be evaluated. Some of the factors to address 
in evaluating community involvement are: 

• interactive communication processes and public information meetings; 

• procedures used to involve community groups in planning, decision-making and program 
operation; 

• day-to-day use of external social service agencies in the community; and 

• participation of private sector employers in evaluating the labor market requirements 
that impact upon the education needed by high risk students. 

There are many other school climate factors unique to an individual program and unique to the 
community environment that may be critical to the success of a dropout prevention program. It is the 
program director's responsibility to identify these factors and to work with the evaluation manager in 
developing the evaluation processesto be used to assess the impact of thesefaciors upon the dropout rate. 

Another important factor that affects the dropout rate among high-risk students is the overall clinute 
of the educational institution. Inservice forteachers and other educational staff persons is also an essential 
component in a dropout prevention program. The attitude of teachers and other staff persons toward the 
high-risk student and toward the special services available forthese students are important factors affecting 
the successful operation of a dropout prevention program and must be evaluated. Some of the variables 
to be considered in evaluating the impact of the dropout prevention program upon the institutional climate 
and upon teacher and staff attitudes are: 

• changes in teacher and staff perception of the probability of the high-risk student completing 
school; 

• cha .ges in teacher and staff willingness to support the special services to be provided for the 
high-risk students; 

• the degree of support provided for the program operation by the principal or other school 
administrator; and 

• the value put upon the program by school staff. 

This list of possible factors to be evaluated when assessing the impact of a dropout prevention program 
upon the educational sta f and the overall climate of a school or community is just a sampling of the possible 
variables that could be examined. The factors to be considered in designing an evaluation plan would be 
developed by a program director and evaluation manager and would be unique to an individual program. 

The evaluation process described in this handbook is generic and can be adapted and used to produce 
an evaluation plan for all components in a dropout prevention program including inseivice workshops for 
staff, employers' evaluation of work tudy students' work habits and the adequacy of their education, the 
effectiveness of the participation of community group volunteers in the program, and the parental attitude 
toward the dropout prevention and their sense of involvement and responsibility tor the program s success. 
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In most cases the evaluation coordinator will need to develop instruments for use in the evaluation 
processes. Sample instruments are provided inthe appendices. The sample instruments provide a starting 
point for developing program specific instruments for a dropout prevention program. 



EVALUATION IN RETROSPECT 

The processes described in this handbook presented the important phases of evaluation. It is basically 
a process used everyday-when considering major purchases such as an automobile, when choosing a 
preschoolforachild.indecidingwhatT.V. shows are worthwatching. All of these daily living choices require 
you to set criteria, ask evaluation questions, collect information make comparisons based on this 
information, and then make an informed decision. When applying the procedures described in this 
wandbook, the evaiuator is applying these everyday common sense evaluation skills to a more complex 
situation using a higher level of sophistication in data collection and reporting. 

Remember, the evaluation of a program is a proven method of validating its success and its value to 
the educational systems, to the community, to the business community, and most important of all, to the 
at-risk students. For a program director there is one cardinal rule to live by— Evaluation: Don't operate 
a program without It. 
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APPENDIX A 

SAMPLE INSTRUMENTS 



This appendix contains sample draft instruments that could be used to evaluate a dropout prevention 
program. 
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SEMANTIC DIFFERENTIAL 



The Semantic Differential uses a set of polar adjectives to measure attitude toward an individual concept. It is 
supported by research as a valid technique to assess attitude. 

The subject or concept addressed in the Semantic Differential is printed at the top of the page and the polar 
adjectives are given below on a 7 point scale with one as the lowest or most negative point. However, on the instrument 
itself don't put all the positive or negative adjectives on the same side 01 the scale. Mix them up so there are positive 
and nep?\tive adjectives on both the right and left side of the page. 

Before scoring the Semantic Differential you must reverse the scale for the negative adjectives on the right side 
of the page. For instance, the scale values in ( ) are the values to be used when scoring this adjective set. 

Glad Sad 

1 2 3 4 5 6 7 

(7) (6) (5) (4) (3) (2) (1) 

Sample instruments are illustrated as Appendices A- 1 andA-2. The list of polar adjectives below can be used 
to develop other instruments. 



Calm 


Anxious 


Smart 


Stupid 


Satisfied 


Frustrated 


Ordinary 


Special 


Careful 


Careless 


Sunny 


Cloudy 


Successful 


Unsuccessfu! 


Exciting 


Boring 
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APPENDIX A-1 

Identification 

Information 
PERSONAL FEELING 

Put an X on the line closest to the word that tells how you feol about yourself. 



Name/ID No. 



School 



Class 



Ordinary 



Special 



1 2 3 4 5 6 7 



Smart 



Stupid 



1 2 3 4 5 6 7 



Careless 



Careful 



1 2 3 4 5 6 7 



Useful 



Useless 



1 2 3 4 5 6 7 



Happy 



Sad 



1 2 3 4 5 6 7 



Warm 



Cold 



1 2 3 4 5 6 7 



Successful 



Unsuccessful 



1 2 3 4 5 6 7 



Sunny 



Cloudy 



1 2 3 4 5 6 7 



Exciting 



Boring 



1 2 3 4 5 6 7 



Fast 



Slow 



1 2 3 4 5 6 7 
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APP'iNDIX A-2 

Identification Name/ID No. 



Date 



Information School 



Class . 



SCHOOL 



Put an X on the line closest to the word that best describes how you feel about school. 



Good 



Slow 



Unimportant 



Easy 



Happy 



Stupid 



Sweet 



Comfortable 



Frustrated 



Exciting 



Angry 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



1 2 3 4 5 6 7 



Bad 



Fast 



Important 



Hard 



Sad 



Smart 



Sour 



Anxious 



Satisfied 



Dull 



Calm 
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SAMPLE SURVEYS 



The student, parent and community surveys are questionnaires focused on specif ic topics or issues about which 
information is needed. The content of such surveys will vary with the characteristics of a program as well as its goals 
and objectives. Appendices A-3 through A-5 illustrate how these kinds of surveys may be designed. 
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APPENDIX A-3 

Identification Name/ID No. Date 

Information School Class 

Student Survey 

Please respond to each item. There are no right or wrong answars. Mark the responses that show your opinion. 
1 . If you want a good job you have to graduate from high school. 
agree not sure disagree 



2. I enjoy going to school. 

never sometimes __ all the time 



If I could choose between going to high school and getting a job, 

I'd stay in school I'd drop out and get a job 

My parents: 

don't care if I finish high school or not want me to finish high school 



The counselor in my school: 

_ doesn't know me at all meets with me only when I'm in trouble 

meets with me only to set up my classes meets with me every week 



6. If I drop out of school my parents: 

will be angry will say it's o.k. will be glad because I'll get a job 
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Finishing high school is: 

stupid smart necessary 

what I want to do not in my plans 
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When I have a problem in school I would get help from: 

my parents the counselor 

my friends no one 

I'm staying in school: 

until I'm old enough to quit as long as I can keep up with the studies 

as long as my friends do as long as my parents make me until I get a diploma 



10. 



I wish I could: 

quit school and get a job 

find someone to help me study 



get better marks in school 

stay in school and still have a job 



11. 



The time I spend in school is: 



a waste of time 



very frustrating 

time I'd get paid for if I had a job helping me get ready for high school graduation 



12. 



Staying in school and graduating: 

will be a way to meet people from the business world 

won't help me get to know the people who give you jobs 



13. 



Briefly describe what you expect to be doing five years from now. 
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APPENDIX A 4 

Identification Name/ID No. Date 

Information School Class 

Parent Survey 

We are asking for your opinion about high school and your role as a parent. Please check the response that best describes 
your opinion or knowledge about each item. 

1 . High school graduation is not needed to get a job. 

agree § not sure disagree 

2. Graduation from high school is up to the student— parents have no part in it. 
agree not sure disagree 

3. You need a high school diploma plus additional training or college to get a job with a future, 
agree not sure disagree 

4. Parents should do everything they can to help their children finish high school, 
agree not sure disagree 

5. Earning money to support yourself is more important than finishing high school. 

* 

agree not sure disagree 

6. If my child wants to quit school, it is the school's job to stop him/her from doing it. 
agree not sure disagree 
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7. tf my child talKed about dropping out of school: (Check all appropriate responses.) 

I'd go to his/her counselor for help 

There's no one at the school I could go to for help 

I'd do everything I could to keep him/her in school 

8. There isn't anything I could do if my child dropped out of school. 

agree not sure disagree 

9. The school and the parent(s) should work together to stop students from quitting school. 
agree not sure disagree 

10. If my child's school had a dropout prevention program: 

I'd support it but I don't have time to go to meetings. 

I'd support it and be actively involved. 

It would be a waste of time 



ERIC 



fil 



45 



The Evaluation Handbook 



APPENDIX A-5 

Identification Name/ID No. 



Date 



Information 



School . 



Class 



Community Survey 

We would like to know what your opinion is about the problem of high school dropouts in our community. Please complete 
this survey and return it to: (person and address) by (date). 



Check all that apply to you, 

Status Asia 

Head of Household 1 8-25 

Registered Voter 26-30 

Business Owner 31-35 

Employed 36-40 

Retired 41-45 

Professional Person 46-50 

, On Public Assistance 51-55 

— Unemployed 
Educational Information 

High School Graduate 

College Graduate 

Some College 

Technical Training 

Trade School 

Not a High School Graduate 

Highest Grade Completed 



Ethnicity 

Black 

White 

Asian 

Hispanic 

Other: (Specify) 



Number of Children 



5-11 
12-14 
15-16 
17-18 
19 & Over 



Single 

Head of Household 
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Married 



Widowed - Divorced 
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High school dropouts are a school problem not a community problem. 

agree not sure disagree 

A high rate of high school graduation will benefit the economy of this community. 

agree not sure disagree 

A dropout prevention program should be part of our local school system. 

agree not sure disagree 

A high rate of high school dropouts costs the taxpayers money. 

agree not sure disagree 

A community-business-school partnership is needed to build a successful dropout prevention program. 
agree not sure [ disagree 

The community, businesses and parents can help prevent high school dropouts by: 

serving in an advisory role for the dropout prevention program 

actively participating in planning the dropout prevention program 

contributing resources such as materials and equipment for special courses 

teaching occupational skills to high school students 

serving as mentors for the at-risk students 

establishing a communications network between school and community 
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EVALUATION QUESTIONS FOR SCHOOL CLIMATE AND 
COMMUNITY ENVIRONMENTAL FACTORS 



This appendix contains broad-based evaluation questions and suggested data collection instruments that 
could be used to collect information on the environmental factors in a dropout prevention program. 
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EVALUATION QUESTIONS AND ELEMENTS TO BE EVALUATED 

1. School Climate 

Question: How do the administration, staff, and students view the Dropout Prevention Program? 
Elements to evaluate: 

• Acceptance of the program 

• Willingness to participate 

• Enthusiasm for program 

• Expectation for success 
Instruments to use: 

• Interview schedule 

• Attitudinal measure 

• Logs of attendance at meetings 

• Logs of participation in program 

• Surveys 

2. Community Involvement 

Question: Is there evidence of a CommunitySusiness-School participatory partnership? 
Elements to evaluate: 

• Contribution of time from community, business and school leaders to the program 

• Participation of business and community persons in district activities 

• Financial and material contributions from community agencies and 
businesses to the operation of the program 

• Accessibility of businesses and the community agencies to program staff and students 
Instruments to use: 

• Surveys and questionnaires 

• Minutes of joint meetings and activities 

• Logs of participation levels 

• Interview schedules 
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3, Program Management 

Question: Was there a detailed management plan for the program that spelled out the lines of authority, 
communication channels, and delegated responsibility for specific tasks with a specified timeline for completion ? 

Element? to evaluate 

• Authority of program manager 

• Scheduling and completion of tasks 

• Recordkeeping 

• Ongoing evaluation of operations 

• Crisis management 

• Problem resolution 

• Financial stability 

• Material procurement 

• Interagency communication 
Instruments to use: 

• Program records 

• Meeting summaries 

• Observations 

• Interviews 

• Activity logs 

4. School Administrators - Staff Relationships 

Question: Did the school administrator play a benign-neglect role or was he/she a supportive, participating leader 
who regularly interacted with the staff? 

Elements to evaluate: 

• Quality and climate of staff meetings 

• Attitude of staff toward administrator 

• Degree of facilitative action on part of the administrator 

• Number/Kind/Effect of administrative delays in program operation 

• Administrator's self-image 
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Instruments to use: 

• On-site observations 

• Interviews with staff 

• Interviews with administrator 

• Staff meeting minutes 

• Records of program operation 

• Analyses of program delays 

• Staff-Administrator questionnaires 

5. Public Relations 

Questbns: Was there a successful public relations effort that involved the community-businesses and school? 
What Wi,s its impact? 

Elements to evaluate; 

• Newsletters 

• Public meetings 

• Brochure(s) 

• Formal presentations of the program to the public 
Instruments to use; 

• Log of public relations production meetings: Who participated? 

• Contribution records supporting community-business-school cooperation in public relations 

• Telephone response to TV-Radio announcements 

• Community surveys 

• Newspaper surveys 

• Telephone surveys 
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6. Curriculum 

Question: How well was the dropout prevention program integrated into the total curriculum ? 
Elements to evaluate; 

• Degree to which dropout prevention permeates the curriculum 

• Staff awareness that the high-risk students must be considered in instructional planning 

• Records of curriculum committee meetings 

• Analyses of K-12 curriculum guides for adaptive materials and activities for the 
at-risk student 

Instuments to use; 

• Review minutes of curriculum committee meetings 

• Review District course descriptions 

• Review District graduation and promotion policies 

• Interview curriculum committee staff 

• Interview parents about student course selections 

7. School Mission/Philosophy 

Question: Is the problem of the at-risk student specifically addressed in the school's missbn statement? 
Elements to evaluate: 

• The language in the school's mission or philosophy statement 

• Relationship between the dropout prevention program goals and the mission statement 

• Actions of the school board 

• Administrative decisions by the district administrators 
Instruments to use: 

• Checklist of critical concepts to be included in the mission statement 

• Checklist of concepts in the dropout prevention program goals for comparison with mission 
statement 

• Administration-Staff questionnaire focusing on the mission stater *nt content relative to 
high-risk students 

• Review of school board minutes 
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8, Program Goals and Objectives 

Question: Do the operating program's activities and procedures reflect the stated goals and 
objectives? 

Elements to evaluate: 

• Records of program activities and procedures 

• Staff perception of program's goals and objectives 

• Administration's perception of program goals and objectives 
Instruments to use: 

• Checklists of goals and objectives 

• Staff interviews 

• Staff questionnaires 

• Administration interviews 

• Administration questionnaires 

9. Social Service Agencies 

Question: Was there official coordination of program services with external social service 
agencies to provide at-risk students access to all resources available for them? 

Elements to evaluate: 

• Ongoing, day-to-day participation of social service representatives in the program 

• The number of eligible at-risk students receiving: 

* Aid to dependent children 

* Free lunches 

* Prenatal care 

* Day Care 

* Transportation 

* Health Services 

* Remediation 

* Psychiatric aid 

* Therapy 
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Instruments to use; 

• Summaries of school referrals to social service agencies 

• Student interviews 

• Parent interviews 

• Summaries of social service agency logs/records of resources provided 

• Summary of day-care use, parents and transportation services 

• Follow-up checklist of medical referrals and services 

• Interview with directors of social service agencies focusing on success of coordination as 
noted in their federal reports 
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APPENDIX C: EXPANDED EVALUATION CHECKLISTS 



EVALUATION PHASE I 



IDENTIFYING EVALUATION ELEMENTS 



Step 1. Review program description, proposal and materials. 



• List the critical program components and elements that appear in the program description, 
proposal and materials. 



Step 2. Analyze the program goals and objectives for each component of the program. 



• Add the critical elements and concepts in goals and objects to the list of Elements to be 
Evaluated. 

• Combine similar or overlapping items on the list. 



Step 3. Prepare a preliminary list of Elements to be Evaluated 



Step 4. Submit the preliminary list of Elements to be Evaluated \o the vested interest groups for reaction, 
expansion and approval. 



• Schedule meetings with representatives of the vested groups. 

• Mail copies of the preliminary list of Elements to be Evaluated \o the vested interest groups. 

• Hold meetings with representatives of the vested interest groups and discuss the Elements to 
be Evaluated, elicit additional information needs of each group and their approval or rejection 
of any of the items on the preliminary list of items to be evaluated. 

(ALTERNATIVE: If a face-to-face meeting cannot be held, have the vested interest groups mail 
in their responses to the preliminary list of objectives as well as their information needs.) 
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Step 5. Prepare the final list of Elements to be Evaluated. 

• Analyze input from the vested interest groups for overlap and duplication, 

• Prepare the final list of elements to evaluate using both the vested interest group contributions 
and the preliminary list of Elements to be Evaluated. 

• Edit the final list for duplication and delete repetitive items. 

• Prepare cnoies of the final list for mailing to the vested interest groups. 

Step 6. Have the final list of Elements to be Evaluated approved by administration and by the vested 
interest groups Involved with the program. 

• Meol with the administration and present the final list of Elements to be Evaluated. 

• Have the administration act to approve the final list for implementation. 

• Send the approved final list to the vested groups for their approval. 

Step 7. Prepare the Summary of Elements to be Evaluated Form. 

• Enter the items from the approved final 1st of elements to evaluate on the form: 



SUMMARY OF ELEMENTS TO BE EVALUATED 


ELEMENTS 
TO EVALUATE 


WHEN TO 
EVALUATE 


PROCESS 
EVALUATION 


OUTCOME 
EVALUATION 
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EVALUATION PHASE II 



GENERATING EVALUATION QUESTIONS 



Step 1. Analyze the program proposal, description and materials as well as the Summary of Elements to 
be Evaluated Form. 

• List questions that should be answered about the program's implementation and operation. 

• List questions that should tell about the impact of the program on students and upon 
environmental factors, 

• Relate each question to the element(s) in the Summary of Elements to be Evaluated 
(Some questions may be related to more than one element.) 

• Review your set of questions for duplication and overlap. 



Step 2. Write the list of questions that should be answered about each of the program's elements, goals 
and objectives. 

• Compare the refined set of questions produced in Step 1 with the program elements, goals and 
objectives. 

• Review all of the evaluation questions for duplication and overlap and delete unnecessary 
questions. 

• Make copies of refined set of evaluation questions to share with administration and vested 
groups. 



Step 3. Submit the list of evaluation questions to decisionmakers and vested groups for reaction, 
expansion and approval* 

• Meet with the administration to present the evaluation questions. 

• Obtain from the administration any additional questions they need answered about the 
program. 

• Revise the list of evaluation questions to include information needs of the administrators. 

• Obtain final approval of the revised set of questions from the administration. 

• Schedule a meeting of representatives of the vested interest groups to review the 
administration-approved questions. 

• Meet with the representatives of the vested interest groups to discuss administration- 
approved questions and to get any additional questions about the program that they want 
answered. 
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Analyze the administration-approved evaluation questions and the additional 
questions from the vested groups fcr duplication and overlap. 

Prepare the final list of evaluations that will be used to drive the evaluation procedures. 

Submit the final list of evaluation questions to the administration for approval and 
then to the vested interest groups for their endorsement 



Step 4. Enter the final list of evaluation questions in the Evaluation Plan Form. 

• Enter the evaluation questions in the first column of the Evaluation Plan. 

• For each evaluation question list the program goals, objectives and elements to be 
answered by that question in column 2 of the Evaluation Plan (see pages 12 

and 13). 



7*. 



erJc» 



The Evaluation Handbook 



EVALUATION PHASE III 

SELECTING A DESIGN, COLLECTING AND ANALYZING DATA 

Step 1. For each evaluation question answer the following Items: 

• How many groups will be in the evaluation Study? 

• What kind of data will provide the answer to each of the evaluation questions? 

• When will data be collected? 
... Posttest? 

... Pretest and posttest? 
.„ Series of tests overtime? 

• Will randomization be used? 

Step 2. Use your answers to the Items In Step 1 and the chart below to select an evaluation design and 
data analysis procedures appropriate for each evaluation question. 



DESIGN 


.v.'.v. -.v.v.'.v. V.'.V. • 


DESIGN 

-■■•3 • 


Vxv; : : : : : :. : :V: ; :'J:\'. : ::: : :|::X : 
x-.v.v.-a-.v 

■.v.'.y.'JI .•.•.•■•:•:•:•.•:•:•■•■•>■• 

: ; : : ; :|: : : : ^;%-: ; : : :v: : :-: ; ;v:>:: 






ONE GROUP Y 


Y 


Y 








TWO GROUPS 






Y . 


Y 


Y 


PRETEST 


Y 




Y 






POSTTEST Y 


Y 




Y 


Y 


Y 


SERIES OF TESTS 




Y 








RANDOMIZATION 






Y 


Y 
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Design 1: One Group Posttest Only 



Data analyses: frequency counts, percentages/proportions, 
mean scores, etc. 



Design 2: One Group PretesUPosttest 



Data analyses: differences in proportion, correlated T-test of 
difference of means, Chi Square 



Design 3: One Group Time Series 

Data analyses: frequency counts, percentages/proportion of 
students at each score level over multiple testing dates, 
correlated T-test of difference in means between first and last 
testing 



Design 4: Pretest-Posttest with Control Group 

Data analyses: Analyses of Variance, Independent T-test 



Design 5: Posttest only with Control Group 

Data analyses: differences in proportion 



Design 6: Nonequlvalent Control Group 

(Control group could be baseline data from school records of similar students.) 
Data analyses : differences in proportion, frequency counts 
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EVALUATION PHASE !V 
INSTRUMENTATION 

Step 1. For each evaluation question, determine from whom data will be collected. 

• Students 

• Staff 

• Parents 

• Business Leaders 

• Work Experience Coordinators 

• Social Service Agencies 

• Community Groups 

• General Public 



Step 2. For each evaluation question list the major categories of data needed. 

• Academic achievement data 

• Count of students meeting defined criterion levels 

• Hours of special services received 

• Student opinion data 

• Data on attitudinal changes 

• Hours of work experience 

• Quality of work performance 

• Participation level of community, businesses and sod* 1 ;rvice agencies 



Step 3. Identify and obtain instruments available to collect the data needed. 

• Check with other similar programs to find out if they have the type of data cottection instruments 
that you need. 

• Contact test publishers and ask for a listing of tests that would be suitable for your purposes. 
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• Search the literature for reports on successful dropout prevention programs for instruments 
used in evaluation, 

• Make a list of all instruments required and order sample copies to examine. 

• Analyze the sample instruments and select the ones that will be used. 

• Order the instruments so that delivery will be at least three months before the testing date. 



• Review the Evaluation Plan and list all the types of instruments listed in the data source column. 

• Compare the data needs listed in the Evaluation Plan with the set of instruments ordered. 

• List all data needs that are not covered by the instruments identified and ordered. Instruments 
will have to be developed to collect this data. 



• Identify staff members who have a working knowledge of the evaluation. 

• Meet with staff persons identified and discuss toe need for program-developed instruments. 

• Train the selected staff persons in item writing. 



• Staff persons write items for each instrument. 

• Review items for ambiguity. 

• If any item or its response set is asking for information about more than one thing, then break 
that item into two or more items. 

• Combine the items into instrument format. 

• Produce draft sets of the instruments. 

• Field test 'he draft instruments by having a small group of persons respond to each one. 

• On the basis of the field test revise the items to improve clarity. 

• Produce the final cooies of the instruments so that they are available at least three months 
before the date of the testing. 



Step 4. Identify any Instruments that must be developed. 



Step 5. Select staff to write Items for the Instruments. 



Step 6. Construct Instruments. 
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EVALUATION PHASE V 



COLLECTING, SUMMARIZING AND INTERPRETING DATA 



Step 1. Examine the evaluation design and set data collection dates. 

• Pretest. . . should be scheduled the week before the program begins but it may have to be given 
during the first week of the program. 

• Posttest. . . should be given no earlier than the last month of the program and preferably during 
the last week of the program. 

• Time Series.., multiple administration of the same instrument should be scheduled over equal 
time periods during the program (every 3 months, every 6 months, first day-midterm-end, etc.). 

• Enter data collection dates into the program schedule. 

• Set in stru ment delivery dates to be su re all in sxruments will be availabl e at least 1 2 weeks before 
the data collection dates set for their administration. (The latest date possible for instrument 
availability is one month before data collection data.) 



Step 2. Set the schedule for evaluation reports. 

• Review the evaluation plan and make a list of all of the evaluation reports needed and their 
delivery dates. 

• Set up a procedure for delivery of data collection results for data analysis. 



Step 3. Train the data collection staff. 

• Sele hire data collection staff. 

• Train the datacollection staff to administer the instruments. This training should be done at least 
one month before the scheduled testing date and should include practice administration of each 
instrument. 



• Train substitute datacollection staff to replace the regular data collection staff in case of illness, 
etc. 
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Step 4. Collect data. 

• Reserve the on-site locations for the test administration. 

• Deliver instruments to data collection sites the week of the test date. 

• Schedule the pick up of the data from the test sites on the day of the instrument administration. 

Step 5. Analyze and summarize the data for each evaluation question. 

• Select or hire the person(s) who will process, analyze and summarize the data for the program. 

• Set dates for delivery of data to the data analysis staff. 

• Set dates for delivery of the data analysis and computer data summaries. 

• Deliver the data to the data analysis staff as soon as it has been collected. 

• Review the data analysis and computer data summaries to locate the data needed to prepare 
the tables and other displays for evaluation reports. 



Step 6. Prepare tables, graphs and charts to display the data summaries needed for each evaluation 
question. 

• Design the format for the data displays to be included in the evaluation report. 

• Use the data analysis and computer summaries to select the information that will be used in 
each table, chart, graph or other data display. 

• Produce the data displays for each evaluation question. 

• For each evaluation question, prepare a narrative explanation of the data displays and the 
implication of the data relative to the program's success. 
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EVALUATION PHASE VI 
THE EVALUATION REPORT 

Stop 1. Write a brief description of the program, its goals and objectives. 

• Describe the program's purpose 

• List the goals and objectives. 

• Summarize the critical features of the program. 

• Briefly describe each major component of the program and its operation. 

Step 2. List the evaluation questions. 

Step 3. For each evaluation question, write a description of the evaluation design, data collection 
procedures, data analysis, data summaries and interpretation. 

• Use the data summaries and interpretations already prepared for each evaluation question. 

• Prepare any additional data summaries and interpretation needed by the audience for the 
evaluation report. 

Step 4. U&e the material written In Step 3 as the body of the evaluation report. Secfirn headings will be 
the evaluation questions. 

Step 5. Write the executive summary of the evaluation report. 

Step 6. Prepare the draft of the total evaluation report and submit to the project manager for review and 
approval. 



Step 7. Prepare the final copy of the evaluation report and turn It over to the project for publication and 
distribution. 
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The Evaluation Handbook is part of a series of dropout prevention research reports published by the 
National Dropout Prevention Center. Additional copies may be ordered. The Center has produced a variety 
of other products which can be helpful to those who work with at-risk youth. 



To obtain a complete list of publications and prices call or write: 

Publications Department 
National Dropout Prevention Center 
Clemson University 
Clemson, SC 29634-5111 
(803) 656-2599 
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